Talk:Disk formatting

/unformat undoing formatting

It's relatively untrue that formatting will cause you to lose all your data, a simple /unformat [drive] will usually restore all of the files. — Preceding unsigned comment added by 67.51.19.86 (talk • contribs) 17:57, 15 September 2005‎ (UTC)

That might be true of what the page calls "high-level formatting", but I doubt it's true of "low-level formatting". In any case, the "Recovery of data from a formatted disk" section says "As in file deletion by the operating system, data on a disk are not fully erased during every high-level format."; I've changed the second paragraph to say "...some of this data might be recoverable with special tools" (I wouldn't recommend reformatting a disk with an arbitrary file system type and expecting all your data to come back easily). Guy Harris (talk) 23:13, 31 May 2012 (UTC)

It's still wrong. The data is still there. You can't have any 'might be true' in this document. The facts need to be culled for this article, otherwise the article has no point. Try looking up EnCase to see how data is recovered. Look into Peter Gutmann's discussion. Data is not lost - it's only unindexed if you're just formatting and no more.

Now fixed with cites (they were the hardest to find :-). Destructive low-level formatting is a thing of past technology such as FDDs and ST506 HDDs. Tom94022 (talk) 20:18, 22 November 2012 (UTC)

Quick Format and NTFS

== == "Never use quick format when formatting a NTFS Drive. There is a chance the drive could become corrupted. Maybe not a big chance, but it is there. Better safe than sorry."

Is there any information/links that back up this claim?

I'm removing it now. No reference was provided, and it doesn't belong in this article anyway. The article doesn't say what either quick format or ntfs means. --Ropez 08:33, 23 April 2006 (UTC)

No unformatting utility can recover data from a partition that was formatted by the /u parameter. This is not the most secure way of destroying the previous data, instead use something like DBAN to destroy old data, however no disk wiping software guarantees 100% destruction of stored data. Only physically destroying the hard drive itself along with the magnetic particles will guarantee complete security. "No unformatting utility can recover the data, but this is not the most secure way of destroying previous data". This sentence is contradictory as it is.--Ricardo Dirani 14:30, 16 May 2006 (UTC)

While I'm no expert in this field, I do happen to know that it's possible to recover data that has been overwritten a certain amount of times due to minute residu of the overwritten data on the disk - that's about the extent of my knowledge for this particular case. So, while software might not be able to recover the data, someone with the knowledge and technology - and actual drive - could recover the data.

Ah, here's a relevant article: MFSTM. Laogeodritt 14:42, 11 June 2006 (UTC)

I would not trust the MFSTM article, as it stands at the moment. See the links I posted in Talk:MFSTM, especially this one [1], and also the arguments discussed in the data recovery article. Mtford 19:59, 15 August 2006 (UTC) == ==

Over writing

I removed this:

", or even better, a low-level format must be performed. This is actually incorrect.

The data can still be recovered through physical means (by consulting data recovery specialists); to prevent this, the drive must be securely wiped, although even this is not a perfect guarantee. Only physically destroying the hard drive itself along with the magnetic plates will guarantee that the data is truly gone, but if you only break it into pieces, the data may still be recoverable if the appropriate method is used"

Anyone who wants this, or similar, put back in, should link to a company, or software, or hardware, that can recover data that has been overwritten. See also Gutmann_method where Gutmann himself debunks the myth about overwritten data being available. DanBeale 11:06, 29 March 2007 (UTC)

Thank you for adding this note, and the link to Gutmann's own Epilogue statement, where he wrote too many have completely misunderstood what he'd written; and at a time when HDDs were nothing at all like they are today! Daniel Feenberg, whose article is listed as criticism of the so-called "Gutmann Method" there, actually e-mailed me and asked I review his paper. Apart from one of the last sentences which seemed to be a bit too personal about Gutmann himself, in general I'd agree with his assessment of the topic. Even if GOV agencies had some of the equipment paranoid citizens believe they do, they simply cannot afford to waste all the time it would take to find a few bits and pieces of data completely out of any context! BTW, the reason Feenberg contacted me was that he'd found my own web page here: http://thestarman.pcministry.com/asm/mbr/WIPE.html "How To Permanently Erase Data from a Hard Disk". Daniel B. Sedory 10:24, 15 April 2007 (UTC)

Oh, I forgot to say this: One thing that really irks me are those who throw about the phrase "low-level format" when speaking of hard disks. Many ignorant people continue to promulgate the idea HDDs should not only have this done to them, but that it's even possible; which it def. is NOT for anyone without access to an HDD factory for any modern (like over 12 years or more now) HDD, so I'm quite happy to see anything like that removed from this article! Daniel B. Sedory 10:43, 15 April 2007 (UTC)

Recovery of data from a formatted disk

In the last paragraph, the article states the DOS FORMAT command completely overwrites every sector when run on a floppy disk by writing F6h bytes to each sector. Each sector of a floppy contains 512 bytes which is 200 in hex. Where did he get F6? F6 = 246 in decimal or 11110110 in binary. 246 is an unusual number as most numbers in computing are powers of 2.

Also, the old dos version of format would only write zeros to the sectors if you used the /u (unconditional) option, otherwise the data could be recovered with the unformat command.

The help files on Windows Vista claim that the format utility deletes all data on the disk, unless the quick format option is selected, in which case it creates a new file table without overwriting the disk. The command line format program also has a /p: option which writes a 0 to every addressable sector on the drive multiple times (you supply the number of passes after the /p:). However bad sectors are still a problem as there is no way to determine if the drive was successfully able to overwrite them. If not, bad sectors could potentially contain an image of data previously stored there. David J. Dreier 18 October 2007.

F6 is the value written to every byte on the disk. Like this : F6F6F6F6F6F6... I will rephrase the sentence in the article.

--Xerces8 (talk) 17:25, 22 November 2007 (UTC)

David, although Xerces8's recent edit, in which he added "byte value" before "F6h", is helpful, the key idea would have been "filling each sector" with 'F6h' bytes; not just writing it once.

Do you know which version of DOS wrote zero-bytes during a FORMAT (and not 'F6' bytes)? Have you performed any tests to verify that? That's something I'm unaware of, but could test in the future as I have access to various versions of DOS from a collector friend. Is there anything on the Net or in a manual that specifies what certain versions of DOS write? I've done many tests with the FORMAT command, but don't recall seeing anything except 'F6' bytes during a FORMAT.

I'd also like to point out for everyone that until VISTA's apparently recent true data deleting function (I've yet to examine that myself), only the formatting of floppy diskettes (and possibly rather small partitions on hard disks), truly wiped all data (that was possible for it to overwrite) from a medium. Performing a FORMAT on large hard disk partitions (of FAT32 and NTFS for example), leaves a great deal of data behind. As the article states, it's only what is OVERWRITTEN that is truly deleted. Daniel B. Sedory (talk) 02:23, 27 November 2007 (UTC)

I've added information about this to the article -- complete with a screen-capture (made myself years ago for an old listserv) explicitly showing even FORMAT /U failing to always overwrite. Note that in my case, the partition was only 8 MB in size, and also, I was running MS-DOS 6.22a at the time, meaning the partition type would have been either FAT12 or FAT16 (can't remember which it was, but since FAT12 supports up to 32 MB, it could have been either). Also note that while I was using NDOS 8.0 as a COMMAND.COM shell replacement at the time, NDOS 8.0 had no built-in or external replacements for FORMAT.COM or UNFORMAT.COM; so the versions of FORMAT and UNFORMAT used in the screen capture werein deed truly MS-DOS 6.22a's. 71.160.20.130 (talk) 09:40, 6 March 2010 (UTC)

F6h is the value typically used for floppies and hard-disks on IBM-PC compatible machines, but other vendors have used other values in the past. For example, 8-inch CP/M floppies typically came pre-formatted with a format filler value of E5h, this was also implemented in Digital Research formatting tools, and thereby this value also found its way to Atari ST and some Amstrad/Schneider formatted FAT media. Amstrad also used a format filler value of F4h. Today, harddisks are sometimes initialized with a value of 00h, wheras FFh is typically used on flash disks.

The BIOS (and DOS) retrieves the value to use from the DPT (INT 1Eh), a data structure typically set up by the BIOS at boot time, but which can be taken over and changed by disk tools later on. Some DOS tools allow to override this value.

While I would like to learn about any other values used historically (if there are more), the most interesting question is: Why F6h specifically (or whatever else)? As far as I remember, this was originally either just an artifact of the low-level formatting process or/and (if it was variable) "a bit pattern which was particularly good to distinguish for early controllers". I remain vague here as (without looking this up) I don't remember the details of FM/MFM controllers after such a long time, but perhaps it's still a pointer good enough for someone else to find and describe the exact technical reason for this value. --Matthiaspaul (talk) 07:15, 2 August 2012 (UTC)

Best Buy Geek Squad

I am a bit confused over this low level format. As per this article, low level format erases all data and this data is irrecoverable. But i had a discussion with the "Geeks" of "Best Buy", who told me that there data recovery software can retrieve up to 600 GB of data (including over written one) from a 120 GB HD even if a low level format is performed. So, which statement should i believe in?

Oy vey. I'd suggest that you take a lot of things Best Buy's "geeks" tell you with a grain of salt. Nothing like this can be done with software alone if data is truly overwritten, ever. Period. It was once possible, with older low-density hard drives, to strip them to their bare metal, mount their platters in special microscope equipment housed in clean room laboratories, and partially retrieve overwritten data. With modern hard drives, recovery of overwritten data is simply not possible -- unless the CIA or DIA knows something, in which case they aren't talking, and thus only the CIA/DIA can do it. Read this. Also, don't forget to sign your additions (with four tildes) in the future.  :) 71.160.20.130 (talk) 10:16, 6 March 2010 (UTC)

why doesn't format wipe all data?

How can it be that a ("non quick") format does _not_ wipe all data? How exactly is the testing for bad sectors done? If it is done by writing data to each sector and then testing if what's been written is the same what has been read, how can it be that any data is not overwritten and retains on the disk? Anyone who can solve this mystery??? 195.135.137.107 (talk) 15:16, 27 November 2007 (UTC)

In windows 'long' format, testing for bad sectors is done by reading each sector, not writing to it. Modern drives include error correction and sector remapping in the drive controller. If you try to read a sector that's failing, the drive will attempt to recreate the missing data (http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction) and write it to a spare sector. It will then update the drive map so that the bad sector is marked bad and there's a pointer to the new 'spare' sector. So simply reading every sector is enough to make the drive 'fix itself'

Also this is a more likely reason why 'zero wiped' drives are still considered sensitive by government agencies. If any bad sectors appears in the life of the drive they're automatically mapped out and never used again. They won't be overwritten if the drive is zero-filled. Any data in those sectors can still be recovered by putting the drive controller into a special diagnostic mode and telling it to directly read the mapped out bad sectors.

118.90.57.194 (talk) 21:23, 14 September 2011 (UTC)

I can't understand one paragraph

Hello, I dont have Englisn as mother tongue. The last sentence: From the perspective of preventing the recovery of sensitive data through recovery tools, the data must either be completely overwritten (every sector) with random data before the format, or the format program itself must perform this overwriting; as the DOS FORMAT command did with floppy diskettes, filling every data sector with the byte value F6 in hex.

Does it mean that the DOS FORMAT can prevent data from being recovered? If I want to keep my data from format-recovery, what should I do? Format it from NTFS into FAT32 then once again back to NTFS?Does it work? —Preceding unsigned comment added by 222.71.122.88 (talk) 04:50, 3 February 2009 (UTC)

"Reformat"

The term "Reformat" would logically mean "to format again", as formed by the suffix "Re-" with the word "format". However, the term "Reformat" is not officially accepted by Merriam Webster's Dictionary, nor by Ask.com's Dictionary. This is possibly due to the term not being any different from the term "Format"; both definitions are the same. —Preceding unsigned comment added by 99.41.247.244 (talk) 21:56, 15 August 2009 (UTC)

Qualified over-general statement, added link, corrected history

I've modified the history section to

• Qualify the reference to the host seeing sectors
• Indicate that sectors are fixed length
• Include links for both CKD and ECKD
• Correct history to reflect that the 3350 was the last drive to directly support CKD and that the disk subsystems after the 3390 continued to simulate CKD.
• Added a reference to support of sector orientation in z/Linux and z/VM. Was this last TMI? Shmuel (Seymour J.) Metz Username:Chatul (talk) 12:56, 29 June 2010 (UTC)

Three levels of formating

This section is to discuss an ongoing dispute over the number of levels of formating and to attemmpt to resolve an edit war. The original article discussed only high level and low level formatting. There are actually three:

1. Low level formatting, typically done at the factory, involving such things as writing timing marks.
2. Preparing the disk for use by the operating system. For a PC that typically involves partitioning. For IBM mainframes that typically involves creating a vlume label and a volume table of contents (VTOC).
3. Preparing a drive, partition or logical drive for use by a file system. For a PC that typically is done by the format utility. For IBM mainframes it may be done in a variety of ways, depending on the access method.

I can provide documentation for each of the above types of formating, and the intermediate type goes back to the 1960's. Shmuel (Seymour J.) Metz Username:Chatul (talk) 23:24, 21 December 2010 (UTC)

Hello - I have accepted a WP:3O request for this page. I note that this request has not been advertised here and can find no ongoing dispute or, indeed, 2nd involved editor. All these circumstances suggest that a 3O request may have been inappropriate and that a "Request for comment" may work better. 3O is only for resolving issues between 2 disputing eds. If a 2nd editor would contact me on my talk page confirming the deadlock and that only 2 eds are involved I'd be glad to help. Note: the request has been removed from the 3O page and I remain willing to help with resolution. Thanks. Redheylin (talk) 00:45, 22 December 2010 (UTC)

Sorry. The other editor started the dialog on User talk:Chatul#Yr Disk Formatting Edit rather than on Talk:Disk formatting, and I failed to note the fact in my request. I've asked Tom94022 (talk) to contact you and to confirm thaqt this is a deadlocked two-party dispute. In the meantime, are there any details or citations that I should add to this discussion? Thanks. Shmuel (Seymour J.) Metz Username:Chatul (talk) 12:53, 22 December 2010 (UTC)

• The terms formatting, low-level and high-level formatting are well established terms of the computer art. See for example Two Formatting Steps The term "intermediate-level formatting" appears to be an invention of Shmuel (Seymour J.) Metz Username:Chatul and as such is WP:OR and not permitted in the article. We agree that formatting of disks has been around since the mid-60s. We agree that the DOS and UNIX worlds distinguish between low level formatting and high level formatting. We agree that that subsequent to "formatting" of a 60s era medium and the combination of "low-level and high level formating" of a DOS or Unix medium the medium is suitable for use by an operating system. Apparently Chatul thinks partitioning somehow corresponds to an intermediate level in formatting, but I think most references would include partitioning as a low-level function as it was done in DOS FDISK. It should be noted that as technology has advanced over time most low level functioning has moved to the factory so that today only partitioning remains as a low-level but rarely performed user activity. Under Chatul's definition FDISK is an intermediate level utility, but in fact in the early days it did both what Chatul characterizes as first and second level functions, but more importantly, I believe FDISK is universally known as a low level utility. Accordingly, I believe the article would more accurately reflect the common usage of the terms as a two level structure with partitioning included as a part of the low level formatting. Since this is more or less how the section was originally written I am going to revert to the original article until this dispute is resolved.

Tom94022 (talk) 22:18, 22 December 2010 (UTC)

As I stated in response to your comments on my talk page, the dispute is about the number of levels, not about the nomenclature of the individual level. The article does talk about low level formatting as being done in the factory, and those functions that I referred to as intermediate are clearly at a higher level than that. I already stated that I have no objection if you wish to use a different term to describe the intermediate level, and using a purwely descriptive term does not constitute OR.

The previous text of the article did not describe fdisk et al as low level. See, e.g., Disk formatting#Transition away from LLF

Subsequent to low and intermediate level formatting the medium may usable by an operating system. In z/OS some access methods do not require high level formatting of tracks prior to writing on them. Shmuel (Seymour J.) Metz Username:Chatul (talk) 23:12, 22 December 2010 (UTC)

OK. First, Tom is correct to say that it's incumbent upon those introducing ideas to provide citations for those ideas. However it would not be appropriate for me to revert as he asks. I note, moreover, that the quality of citations in this article is poor overall. Google hits, blogs, etc. should not be necessary in an area that is replete with high-quality documentation. Chatul is correct that the process of HD formatting involves several stages and that classification of HL and LL may be insufficient, and that the terminology used may not be absolutely consistent. Still, statements entered should ALWAYS be sourced to authoritative works.

Looking at an old text-book ("Operating System Concepts" by Peterson and Silberschatz) I find that NO mention is made of the term "format". Even picking up The Waite Group's "Tricks of the MS-DOS Masters", the command "FORMAT" is defined as "disk initialisation". I think it is fair to conclude that the term "format" is, to start with, a Microsoft term, yet this is not clarified here. I'd like to suggest that both of you co-operate in building up this article from the most basic level (low level reformat!) using only referenced material. I think, if you will do this there will be a better article and fewer problems. The problems are caused by accepting terminology and concepts without tracking their source and history and the result is a poor article. I think if you work together from the best texts you can you will find the problem has just disappeared along the way. OR articles invite further OR. Any comments? Redheylin (talk) 00:53, 23 December 2010 (UTC)

FWIW, the term format and the utility FDISK I think go back at least to CPM. The more common term may be "initilize."

FWIW, Appendix B to the DEC RT-11 System Generation Manual (July 1976) is entitled "Formatting the RK05 Disk." The procedure looks like a "low-level format" not dissimilar from the BIOS process used by PCs but it produces a disk "now formatted and ready for use". I cite this to show the term predates Microsoft and this low-level process apparently does a low-level format and a high level format to use terms of the art. This makes both Chatul's and my text inaccurate. Tom94022 (talk) 02:16, 23 December 2010 (UTC)

Since Chatul doesn't care about terminology I retitled the section and changed "intermediate-level formatting" to "Partitioning" which should obviate both our objections. There still are problem with the Disk reinitialization section which is where we deal with the idea that partitioning either is (my view) or is not (Chatul's view) a part of low level formatting. Tom94022 (talk) 01:35, 23 December 2010 (UTC)

That would be fine for PC's, but it violates NPOV, since the intermediate level of formatting on mainframes does not involve partitioning. Can you come up with a correct and reasonably short term instead? Thanks. Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:38, 23 December 2010 (UTC)

It also applies to UNIX. I have to research and think about it some more put to a certain extent partitioning in the single partition context may be an accurate description of what you call "intermediate level formatting" for IBM mainframes; that is recording into the blocks information that allows the OS to recognize one or more logical disks. BTW, do you have any evidence that the rest of the BUNCH did it this way. This is why I prefer two steps, low-level being all things that make the disk available to the OS and high level being all things that make the disk usable by a file system. How this is done varies over time with technology and by OS but fundamentally it seems to me to be a two step process. Unfortunately there does not seem to be any published material generalizing the subject of "formatting" so anything we do is more WP:OR than WP:NPOV neither of which should be in an article. Perhaps the only way out is to not generalize and rewrite the article into two or three sections, one on PC and UNIX, a second on IBM mainframes and possibly a third on Apple (I suspect it is PC/UNIX like)? Tom94022 (talk) 19:43, 23 December 2010 (UTC)

Both of you seem to be missing the point. There's no point in telling ME HERE about things, particularly when you go on speaking as though you were the experts of the world. It's not a boxing match, there are no awards. Do you all need to be told how clever and expert and RIGHT you are? Write a blog: wikipedia is for collecting the statements of published experts. You are both acting disruptively. I am asking you both please to stop edit-warring and work through the whole article, substituting statements from verifiable and authoritative sources. Redheylin (talk) 23:51, 23 December 2010 (UTC)

The article already discusses partitoning, which falls between what the article calls low level and what the article calls high level. Shmuel (Seymour J.) Metz Username:Chatul (talk) 00:37, 24 December 2010 (UTC)

Actually u, Redheylin, are both missing the point and really acting outside your supposed role to arbitrate a deadlock. First of all my remarks are directed at Chatul and any other editor who can contribute to the discussions; I take Chatul's in the same spirit. Your sarcasm and Ad hominum abuse serve no purpose and are inappropriate to Wikipedia discussion. Secondly, there is nothing disruptive about this discussion and in fact it seems to be arriving at an agreement. Thirdly, it is not edit waring since most of it is fairly expert discussion of the relevant facts on a Discussion page - almost by definition not edit warring. In conclusion, IMO your conduct is inappropriate and if you don't have anything useful to say then you might consider keeping silent. Tom94022 (talk) 02:20, 24 December 2010 (UTC)

How about calling the intermediate level Preparing for OS use? I don't see how using a descriptive term is OR.

The problem with the descriptive phrase Preparing for OS use is that it appears to me to mean every thing done to make the disk available to the OS, which would include both the placing of the appropriate blocks on the disk and writing into them that information which makes the disk accessible to the OS, that is, in the current terms of the article both "low-level formatting" and "partitioning." BTW even today's IBM systems might require a low level format if for example one wanted to install an HDD formatted in one block size into a subsystem requiring a different block size, although admittedly this is an unusual event. I am going to make some minor changes to the section while we work this out. Tom94022 (talk) 02:20, 24 December 2010 (UTC)

I used terms such as IBM mainframes because I did not know how the Seven Dwarves, or the other vendors, handled it. I'd certainly welcome anybody from, e.g., XDS, who could add that information.

The original article discussed factory formatting, which is at a different level from partitioning, ICKDSF et al.

I like the idea of adding sections for various platforms, if we can find editors with experience in, e.g., CDC.

There is published material that describes specific partitioning and other intermediate levels of formatting; I can provide online links for the PC and IBM mainframe worlds. Would it be appropriate to provide only a recent reference for each, or also older references? Shmuel (Seymour J.) Metz Username:Chatul (talk) 00:37, 24 December 2010 (UTC)

Reversion without discussion is edit warring - so is canvassing others to revert - and edit-warring is always disruptive. And in this case it is futile, because your work is sure to be overwritten by an editor who uses available and notable sources and if anyone reverts this they will probably be blocked in the end. Many times it happens that the person who is the most "right" is also the most aggressive - then that person is blocked, right or not. Chatul writes: "I can provide online links for the PC and IBM mainframe worlds. Would it be appropriate to provide only a recent reference for each, or also older references?" At the moment, Chatul, simply add the reference you consider the most authoritative and permanent. Good online sources - NOT blogs or hobby sites - will be very valuable. You must then accept contradictory sources with good references alongside these, but may remove or revert any original research. Then there is no need to seek experts in CDC and other platforms - just find the sources and add them: it is better than "experts" who rely upon their own expertise. "Google Scholar" and "Google Books" are there to use. Copy author - title - publisher - date - page and add the link, and this will be defended. Original research and editorial knowledge will soon disappear and, as I say, you or anyone may be blocked if you resist it. Nobody wants that. Redheylin (talk) 06:06, 25 December 2010 (UTC)

Redheylin did you happen to notice that these issues are discussed on this and other discussion pages? If you read WP:Edit warring you will see that we are not edit warring and that we are not near a WP:3RR situation. I'm not sure what your "canvassing" remark refers to, but if you mean my request that you as the arbitrator revert to the original until a consensus is reached, that is not in any sense "canvassing." So please stop your preaching and Ad hominem attacks. Tom94022 (talk) 18:15, 25 December 2010 (UTC)

I have been unable to find any publication that generically discuss the subject of Disk Formatting so the following is WP:OR but it does appear to me that such a generic treatment would identify four levels of formatting as follows:

1. The first and optional level is placing positioning information on the medium. This is not done on FDs nor was it done on early HDDs but is now done on all HDDs and has always been done on optical disks. The position information may be recorded as in HDDs or permanently embedded as in optical. This is always a factory process.
2. Organizing the disk surfaces into blocks. Depending upon step one the block organization may be limited or even dictated by the nature of the position information. In early HDDs (i.e., any drive such as a CKD drive not using sector servos) and FDs there are no such limitations. HDDs have evolved from a somewhat strict limitation to technically no limitations (headerless split field recording) but a strong de facto limitation due to the ubiquitous 512 byte block size. Optical disk drives have always been severely limited. Sometimes this is a factory process but it can be a user process.
3. Writing information into the blocks sufficient to allow the medium to be recognized by the OS. This can be a factory process but there is always a user way of doing this.
4. Writing information into the blocks sufficient to allow the medium to be used by at least a file system. This is always a user capability and is almost always a user process but in some high volume situations can be a factory process.

The problem we have is that the article started with a strong PC HDD and FDD orientation that is incomplete and even inaccurate when considered in light of other OSes and other media. Chatul and I have been trying to incrementally improve the article without totally rewriting it but the reality of the subject is the absence of a generic publication means its either going to be bad or there will have to be some expert input. If anyone can find such a publication that would really help Tom94022 (talk) 04:29, 26 December 2010 (UTC)

"I have been unable to find any publication that generically discuss the subject of Disk Formatting" - in that case the subject of the page has no notability and should be erased. Redheylin (talk) 08:25, 26 December 2010 (UTC)

The absence of a generic treatment of the subject does not mean the subject is not notable; it's notability is established by reference one, for example. Tom94022 (talk) 17:57, 26 December 2010 (UTC)

The ambiguity in the term “low-level format” seems to be due to both the inconsistent documentation on Web sites and belief by many users that any process below a “high-level (file system) format” should be called a low-level format. Instead of correcting this misconception (indicating clearly such a process can not be performed on specific drives), drive manufacturers have actually described various software reset as LLF utilities on their websites. Because users generally have no way of determining the difference between a true LLF and reset (they simply observe the results of running software on a hard disk to be partitioned and “high-level format”) As the user misinformed and mixed-signal various drive manufacturers have perpetuated this error. http://unix.org.in/2010/10/disk-formatting/ Redheylin (talk) 11:18, 26 December 2010 (UTC)

Your quote above is from a section labeled in its article "This section needs additional citations for verification" so it hardly qualifies as a reliable source. It is also almost verbatim the Disk reinitialization section of this article. The good news is the UNIX article describes UNIX disk formatting as a two step process. The bad news is that it is a UNIX article and it reads very much like this Wikipedia article (including the quote above) so it is not clear who is copying whom and again we wind up with a question of is it a reliable source. Tom94022 (talk) 18:07, 26 December 2010 (UTC)

What is ICKDSF

I started this as a separate thread from the number of levels of formatting so that we can discuss the IBM mainframe approach in one section.

I downloaded (free) and am looking over Release 17 of ICKDSF User’s Guide and Reference to see how it describes its process of making a disk available to a system. At first glance it appears to me that on native CKD devices it performs both "low-level formating" and making the volume (device) available to the system. I will elaborate after I've done a little more reading, but if anyone has any comments please add them. Tom94022 (talk) 05:48, 24 December 2010 (UTC)

FWIW

Part 2 of Release 17 of ICKDSF User’s Guide and Reference "Using ICKDSF to install and maintain CKD devices" , Chapter 18. INSTALL command—CKD, states in part, "INSTALL command is an enhanced installation procedure that includes the writing of home address and record 0 on every track of a 3380, 3390, and 9345 volume." In the terms of this article the INSTALL command is doing a "low level format" since it creates the timing marks associated HA and RO and furthermore does a part of "partitioning" in that it writes information into the RO and HA field necessary but not sufficient for MVS access. The INIT command (Chapter 16) then completes the "partitioning" process by writing "the volume label and VTOC on volumes for use by MVS or VSE operating systems" there by completing the process of making the disk drive medium available to the OS. While this utility doesn't deal with CKD drives prior to the 3375, I am sure this is the process used for all CKD drives back to the 2311.

Part 3 of Release 17 of ICKDSF User’s Guide and Reference "Using ICKDSF to install and maintain FBA devices" does not appear to deal with changing the physical block size (as best I can tell it requires a 512 byte physical block) and does not use the INSTALL command.

Personally I think the CKD process performed by ICKDSF supports my view that their is a one "low level" process which happens to have two steps, blocking (the INSTALL command) and volume recognition (the INIT command). The FBA process of ICKDSF is only the volume recogintion part of "low-level" process; the blocking is done in the factory or by a SCSI or IDE utility.

I also note this apparently contradicts the statement that ICKDSF and other such utilities cannot recover lost timing. In fact, on any dedicated servo surface HDD as on FDDs the partitioning or even the formatting utility could and did handle low level functions such as timing marks. That's why they were called soft sectors. Tom94022 (talk) 20:17, 25 December 2010 (UTC)

I'm sorry that I didn't get back to you earlier; verizon killed my local loop while handling a service call for my neighbor, and they took their own sweet time about fixing it.

The INSTALL command of ICKDSF does not write timing marks; HA and R0 are at a higher level. In fact, INSTALL is used for disks that require factory formatting, not for the older disks that could be fully initialized on site. Note the statement "Tasks you can perform with ICKDSF". Device Support Facilities (ICKDSF) User's Guide and Reference R17. Note: The surface checking functions performed by ICKDSF are not equivalent to the surface checking that is performed on a volume at the factory. For more information, see Appendix E. Surface checking. {{cite web}}: Unknown parameter |separator= ignored (help)

The most recent version of the ICKDSF documentation seems to be IBM. Device Support Facilities (ICKDSF) User's Guide and Reference R17 (PDF). GC35-0033-36. {{cite book}}: Unknown parameter |separator= ignored (help) HTML version; that should go in the article. If . Shmuel (Seymour J.) Metz Username:Chatul (talk) 19:45, 30 December 2010 (UTC)

Welcome back. When an IBM system issues a WRITE HA or WRITE RO to a CKD HDD (and for that matter any other CKD write command) the control unit writes Address Marks and/or Sync Bytes in gaps all of which are "timing marks" and function in just the same way as servo marks and sync bytes do in modern FBA HDDs. The contents of HA and RO may be at a higher level but in the process of writing either, the control unit writes "timing marks," that is does a low level format and in the process fills in the information necessary for volume recognition. The fact that one program, INSTALL of ICKDSF, simultaneously does two disk formatting "functions," i.e. low-level formatting and partitioning, doesn't change the fact that there are two functions. The Address Mark indicates the start of a CKD record, that is, "indicating the start of a recording block" as the phrase is now used in our definition of "low level formatting". Likewise the sync field that locates the start of each CKD field is also a timing mark. Doesn't the DOS Format of an FDD do all three disk formatting "functions" in one program - and in one pass? So it seems to me that certain channel commands, particularly WRO and WCKD "low-level format" the track using our current definition of the disk formatting process. Please explain why not. Tom94022 (talk) 21:06, 30 December 2010 (UTC)

I need to track down and download some old manuals, but since the 2314 IBM has been using servo tracks that cannot be formatted by the customer. There is also some formatting in modern DASD subsystems that can be requested from the console but not by channel commands. The latter is the norm for subsystems that simulate 3390's on arrays of SCSI disks. Shmuel (Seymour J.) Metz Username:Chatul (talk) 18:15, 2 January 2011 (UTC)

I know of no data track formatting in any CKD drive (or SMD drive for that matter) that could not be implemented across the "channel". All IBM CKD drives from the 3330 thru the 3390 used a dedicated servo surface. All data surfaces were fully recordable by the user using a utility of one sort or another. From a formatting perspective, the servo surface acts like a clock track, not substantially different in function from the clock track on a drum, that is, it provides the index mark and writing clocking. The servo surface write clocking is why gap 3 & 4 in these later drives does not have to take into account the speed tolerance - the write clock is synchronous to the spindle rotation. The content of some of the fields (e.g. ECC field or Sync Byte) were not user controllable but the user could cause them to be written and overwritten by channel commands. But the fact that the user could not rewrite the servo surface is no more relevant than the fact that a user could not make an index mark. What is relevant is that we agree that the placing the timing that marks the start of a block is "low-level formatting" and that was a user capability on all CKD drives, all SMD drives, most MFM drives, etc. It is the advent of embedded servos that changed this. Tom94022 (talk) 05:09, 3 January 2011 (UTC)

The point is that low level formatting includes writing the servo tracks, and there is on CCW to do so, either explicitly or as a side effect. That's very different from something that the user can do as part of a more complex operation. Shmuel (Seymour J.) Metz Username:Chatul (talk) 12:55, 3 January 2011 (UTC)

I'm not sure that anyone would agree with you that "low-level formatting" includes writing of servo tracks but even if it does it is only a step in the process of low level formatting using the definition in this article. A 3336 disk pack with just a written servo surface does not have any timing marks that define the location of blocks of data or for that matter any other information; however, every computer system that used those packs provided a utility that then "marks the surfaces of the disks with markers indicating the start of a recoding block ... and other information ..." that is, completes the low level formatting process as we define it in this article. In the days of the 2314 it was all user invokable, today it is less so or not at all. The fact that some parts are user invokable and some are not is only worthy of note but it really hasn't changed anything. Personally I think writing of the servo information is a separate step and prior to low-level formatting but that doesn't change the fact that the INSTALL command of the ISCKSF utility (or earlier variants thereof) does at least a part of the low-level formatting process of all CKD drives and all of the low-level formatting of early CKD drives up to and including the 2319. Tom94022 (talk) 23:47, 3 January 2011 (UTC)

Block CRC written during the formatting process?

The block CRC depends on the contents of the block, so, while each block might be written out with a CRC when a disk is formatted at the low level, the CRC written at that time won't be the block's CRC forever - when the block is rewritten a new CRC will be written. Perhaps some other information should be chosen as an example of what's written on a low-level format. Guy Harris (talk) 23:02, 31 May 2012 (UTC)

Steps to format a disk

It is a bit disingenuousness to assert that a process may take as many as five steps "a half a dozen or more" without providing an example. In its absence I suspect most processes more than three steps could be reduced to the three generic steps stated. Unless someone comes up with an example to support the edit or we get some discussion here, I suggest we delete the edit. Tom94022 (talk) 22:43, 18 December 2012 (UTC)

It's disingenuous to claim that a clarification without an example is disengenuous. I believed that giving an example would be TMI, especially since the example would be unclear without considerable explaination of CKD and VSAM facilities. The case that I had in mind is in the zFS for z/OS. Formatting for zFS requires all of

• Factory formatting of the disks.
• Definition of logical volume to the DASD subsytem
• Formatting of the volume with ICKDSF
• Allocation of an ESDS for the zFS, which entails formatting it into Control Intervals
• Formatting of the zFS within the ESDS

Only after all of the above have been done do you have a zFS that you can mount. Shmuel (Seymour J.) Metz Username:Chatul (talk) 15:52, 27 December 2012 (UTC)

Thanks for an illustration. BTW, five is not "a half a dozen or more" but I suggest the five steps u have identified fit the three processes as follows

1. Low-level formatting == Factory formatting of the disks.
2. Partitioning == Definition of logical volume to the DASD subsytem AND Formatting of the volume with ICKDSF
3. High-level formatting == Allocation of an ESDS ... AND Formatting of the zFS within the ESDS

I going to change ""steps"" to "processes" and change yr note accordingly. Tom94022 (talk) 19:30, 27 December 2012 (UTC)

I haven't checked the history, but I recall five being my original wording and half a dozen being someone else's change.

I have a problem with your classification of ICKDSF as partitioning; it better fits high level formatting, in that it writes the data structures needed by the OS. Also, ICKDSF supports several different operations, and you might not want to categorize all of them the same.

A question that I have not researched is whether any DASD subsystem requires formatting of the physical disks prior to allocating logical volumes on them.

If you do assign those five steps into 3 processes, you should note that there may be more than one type of each process type and that they may be interspersed. Also, you may want to relegate some material to footnotes.

Would you like references for ICKDSF, HFS and zFS? Shmuel (Seymour J.) Metz Username:Chatul (talk) 21:20, 4 January 2013 (UTC)

I have no idea what your recent edit. "and steps of different processes may be interleaved" means so I would appreciate some comment here before I modify or remove it. Tom94022 (talk) 21:40, 13 January 2013 (UTC)

What you described as proceeses are actually categories. The steps for preparing a file system for use need not take place in the order suggested by lumping them into those categories. As ane example, to use the CMS file system you must, in this order,

• Define a logical disk to the DASD subsytem
• Initialize the logical disk with a volume label
• Define minidisks to CP on that logical disk
• Initialize the minidisk for use by CMS

Note that actions you describe as partitioning are intersperesed with actions that you describe as preparing the disk for OS use. Shmuel (Seymour J.) Metz Username:Chatul (talk) 20:07, 15 January 2013 (UTC)

It seems to me the first three bullets are Partitioning (or Volume Creation) and the last bullet is High-level Formatting. Tom94022 (talk) 22:47, 15 January 2013 (UTC)

The breakdown is a bit complicated. Some relevant ICKDSF command differ in their place in the hierarchy; the following ignores some irrelevant details

• CPVOLUME Creates data structures on disk needed by CP, and is done after INIT.
• INIT Creates a volume label, and is done after INSTALL for relevant devices. Note that it also creates a volume table of contents (VTOC), which puts it in the High-level formatting category.
• INSTALL is used to do basic setup for new media on certain devices. Shmuel (Seymour J.) Metz Username:Chatul (talk) 00:48, 17 January 2013 (UTC)

Just out of curiosity, which of those steps involve writing stuff to the disk (I'd consider those steps part of "disk formatting") and which of those steps don't (i.e., steps that involve telling software about stuff that's been done to the disk; I wouldn't consider those part of "disk formatting")? "Initialize the logical disk with a volume label" and "Initialize the minidisk for use by CMS" sound as if they write to the disk; do "Define a logical disk to the DASD subsytem" and "Define minidisks to CP on that logical disk" do so? "Define minidisks to CP on that logical disk" might, as I presume CP stores the minidisk information somewhere in stable storage, which presumably means somewhere on disk. Guy Harris (talk) 22:01, 15 January 2013 (UTC)

The mechanics of defining logical devices to the DASD subsytem vary from vendor to vendor. The other steps always involve writing data to DASD. Shmuel (Seymour J.) Metz Username:Chatul (talk) 00:48, 17 January 2013 (UTC)

Also, the article says "Partitioning creates data structures needed by the operating system", so "partitioning" is, in a sense, "preparing the disk for OS use". As described in the article, though, both partitioning and high-level formatting create data structures needed by the operating system; partitioning writes a volume label/partition map/whatever the on-disk data structure indicating where partitions start and end is called, and high-level formatting writes, typically, file system data structures, and both of those are needed by the OS.

The volume table of contents (VTOC) is certainly part of the file system for OS/360 et al. Shmuel (Seymour J.) Metz Username:Chatul (talk) 00:48, 17 January 2013 (UTC)

Does a "volume" in the OS/360-and-successors sense always correspond to a single DASD? If not, "volume" in that sense could somewhat correspond to the notion of a file system in, for example, the UN*X sense (I think the catalog covered multiple volumes, but that might have been somewhat equivalent to using mounts in UN*X - or in newer versions of NT - to join separate directory subtrees into a single tree). "Volume" in the sense in which I used it when I referred to a "volume label" is a partition map rather than a file system data structure. Guy Harris (talk) 02:06, 17 January 2013 (UTC)

A volume in that sense is anything that can be accessed by the channel subsystem at a single address; for purposes of this thread I'm talking only DASD volumes and not addressing the case of removable volumes. IBM mainframe software requires that DASD volumes be initialized with a volume label and a volume table of contents.

A complicating factor is that volumes can be defined at several different levels in a hierarchy. As an example, a DASD subsytem can spread a volume across several physical drives and can define multiple volumes on a single physical drive, while the CP component of z/VM can subdivide a volume as defined by the DASD subsytem into multiple minidisks.

A volume recognized by z/OS can contain multiple Unix file systems, and z/OS can also spread a Unix file system across multiple volumes. In the case of HFS formatting is done as part of allocation, while in the case of zFS allocation does one level of formatting and a separate utility program does additional formatting on top of that. In neither case is there a quick format option; the formatting involves writing over the entire file system.

There is a similar hierarchy issue with Linux servers, in that you might use an LVM to define logical volumes on top of the volumes provided by the DASD subsytem, which are themselves virtualized. Shmuel (Seymour J.) Metz Username:Chatul (talk) 19:38, 22 January 2013 (UTC)

I'll look at cleaning the article up to better describe partitioning (i.e., describe it less vaguely than "[creating] data structures needed by the operating system"). Guy Harris (talk) 22:12, 15 January 2013 (UTC)

How about replacing Partitioning with Volume Creation or Volume Management, that is, creating one or logical volumes from one or more disks. Partitioning, a term I think from the Microsoft world is one form of volume creation which creates one or more logical volumes in one disk. In the Unix world a logical volume can span disks and of course there is RAID which is frequently a many to many mapping. Tom94022 (talk) 22:47, 15 January 2013 (UTC)

Partitioning might be a term first used in the Microsoft world, but it's certainly used elsewhere. Apple has a pre-OS X tech note that uses the term, and the 4.2BSD HP(4) man page, as unpacked from this tarball, also uses the term.

Creating a logical volume out of multiple partitions may write data structures to the raw disk(s) (if they're not merely specified by an OS configuration file), but that goes above the level of doing something to a single disk. That, however, raises the question of whether "high-level formatting" should be treated as disk formatting at all, given that it could write to a partition that covers the entire disk, a partition that covers part of the disk, a logical volume made out of multiple parts of the disk, or a logical volume made out of parts of multiple disks.... Guy Harris (talk) 04:45, 16 January 2013 (UTC)

I think the reason we are struggling with the semantics in this article is that we have tried to generalize from the rather specific ST506 type HDD disk formatting processes, specifically BIOS low level, FDISK partitioning and Format high level formatting. Unfortunately, I think that takes us into original research. I've looked for generic material on file systems for citation, but without much luck. Most stuff seems to be about this or that file system but nothing treats file systems and the associated formatting of disks as a generic topic. The books by Tannenbaum Ch 6 and Nutt Ch 13 might work so I ordered some used copies from Abebooks. In the meantime, it appears to me that for the purposes of this article there might not be a substantive distinction between a partition and a logical volume, that is, generically the three processes necessary to make some or all of a disk usable to a computing system are:

1. Low level formatting which makes the physical disk drive appear as contiguous blocks of data, today all good.
2. Logical volume drive creation which makes all or parts of one or more disk drives visible to the OS
3. High level formatting which makes the logical volume drive visible to the file system.

Right now I guess this is my WP:OR but hopefully I will find a secondary source that we can reference in the article. The other approach might be talk about each specific instance, starting with the FDD/ST506 example and then comparing and contrasting with some other common examples, i.e., Optioal, Modern HDD, Linux, etc. The latter sounds like too much work. Tom94022 (talk) 19:22, 16 January 2013 (UTC)

Alternate use of logical drive replacing logical volume to avoid ambiguous term. Tom94022 (talk) 19:00, 19 January 2013 (UTC)

I wouldn't use the term "logical volume" there, because, in at least some OSes, disk partitioning and logical volume creation are done at different layers - logical volumes are built from disk partitions. See the logical volume page. That page refers to "physical volumes", which would correspond to partitions (or to full disks; they also speak of SCSI LUNs, but, as the logical unit number page indicates, when talking about disks, those tend to correspond to single disks, otherwise when talking about other random-access media they correspond to, err, umm, logical volumes on a RAID array, where the logical volume is managed by the RAID controller rather than the OS - "hardware RAID" vs. "software RAID").

(BTW, we also should avoid recentism; UN*X and Windows NT may happen to do this one way, as did some older OSes, but traditional mainframe OSes (and even at least some generations of minicomputer and mainframe hardware) may do it in a fashion sufficiently different that a little careful work might be needed to have a general model that describes both, as per, for example, some of User:Chatul's comments. And, yes, various UN*Xes are mainframe OSes; that's why I said "traditional".) Guy Harris (talk) 20:20, 16 January 2013 (UTC)

Should we mention the slice nomenclature used in Solaris? Shmuel (Seymour J.) Metz Username:Chatul (talk) 00:48, 17 January 2013 (UTC)

We seem to agree that a logical volume consists of one or more partitions (aka slice or physical volume) of one or more disk drives where any partition may be all or a part of a physical drive. Partitioning would be an optional step in the process of logical volume creation. In the simplest case, the logical volume == physical volume == physical hard disk drive. Again this is WP:OR unless one of us can find a reliable secondary source. Tom94022 (talk) 03:44, 17 January 2013 (UTC)

Actually, I think that while, in some sense, on a computer running an OS with no logical volume manager, a single partition could be thought of as a "logical volume", I think that referring to it as a logical volume could confuse people used to the notion that a "logical volume" is something implemented by logical volume manager software, and therefore that using the term "logical volume" as if it applied to all OSes would confuse more than it clarified. Perhaps it's unfortunate that the term "logical volume" has become associated with concatenated/striped/etc. groups of regions of disks, but I have the impression that it has become associated in that fashion. Guy Harris (talk) 07:48, 17 January 2013 (UTC)

I note the logical volume management has no reference to its generic discussions but seems to have fallen into our trap of trying to generalize from the specific so I don't put much credence in the article as a source of association. For example the MS-DOS Encyclopedia ©1988 (MS-DOS 3.3) includes the sentence, "For several reasons, large physical block devices such as fixed disks are often logically partitioned into smaller logical block devices ..." and it then was the VOL command that "displays a disks name or volume label." Likewise, the 1999 Microsoft dictionary defines partition as, "A logically distinct portion of ... a storage device that functions as though it were a physically unit" and a logical drive as "A device named by the logic of a software system, regardless of its physical relationship to the system." So, at least according to Microsoft, it is not a stretch from device to volume and that generically, a partition is a logical volume. I am looking for at least one more non-Microsoft reliable source for this subject and when I find it, I will try and fix this article as well as the logical volume management article. Thanks for pointing out the overlap Tom94022 (talk) 18:08, 19 January 2013 (UTC)

FWIW, Modern Operating Systems, 2^nd Editon, A Tanenbaum, ©2001 section 3.4.2, Disk Formatting defines the following generic three step process:

"Before a disk can be used, each platter must receive a low-level format done by software.

...

After low-level formatting is completed, the disk is partitioned. Logically each partition is like a separate disk.

...

The final step in preparing a disk for use is to perform a high-level format of each partition separately. ..."

Tanenbaum did not cover more complex systems such as UNIX, where the logical volume might span physical disks, nor RAID, nor complex data management but it does appear to be a reliable source for a three step process. Using Logical drive creation with explanation for process 2 instead of Partitioning seems, well, logical :-) without being OR. Tom94022 (talk) 22:21, 28 January 2013 (UTC)

Document format for citations?

A number of IBM manuals are available online in multiple formats, e.g., BookManager, HTML, PDF. I assume that most Wiki readers can handle both HTML and PDF, but which is preferable in citations. Shmuel (Seymour J.) Metz Username:Chatul (talk) 01:18, 19 May 2013 (UTC)

"most Wiki readers can handle both HTML and PDF" If by "readers" you mean "humans reading the article", that's likely to be true, although they might have to have downloaded Adobe Reader if they're running on Windows. However, some PDFs might not pop up in a reader if clicked on, in which case the human would have to open up the download directory and open it.

"but which is preferable in citations" All other things being equal, HTML is preferable (more likely that you can link to the specific part of the manual, might render faster), which is presumably why it's the default format for citations (although templates such as {{cite web}} might infer PDF from the extension in the URL. Guy Harris (talk) 01:34, 19 May 2013 (UTC)

U might want to read Wikipedia:Citing sources which emphasizes the content of a cite with no recommendation on content of any link. I do agree with Guy Harris that if a link is available in multiple formats, the preferred order for the three u mention is HTML, pdf, BookManager. Tom94022 (talk) 04:51, 19 May 2013 (UTC)

External links modified

Hello fellow Wikipedians,

I have just modified 3 external links on Disk formatting. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:

• Added archive https://web.archive.org/web/20110519054645/http://www.seagate.com/ww/v/index.jsp?locale=en-US&name=advanced-format-migration-to-4k-tpc&vgnextoid=746f43fce2489210VgnVCM1000001a48090aRCRD to http://www.seagate.com/ww/v/index.jsp?locale=en-US&name=advanced-format-migration-to-4k-tpc&vgnextoid=746f43fce2489210VgnVCM1000001a48090aRCRD
• Corrected formatting/usage for http://freepctech.com/pc/001/007.shtml
• Added archive https://web.archive.org/web/20101129180307/http://seagate.com/docs/pdf/whitepaper/tp595_building_faster_more_flexible_infrastructure.pdf to http://www.seagate.com/docs/pdf/whitepaper/tp595_building_faster_more_flexible_infrastructure.pdf

When you have finished reviewing my changes, please set the checked parameter below to true or failed to let others know (documentation at {{Sourcecheck}}).

This message was posted before February 2018. After February 2018, "External links modified" talk page sections are no longer generated or monitored by InternetArchiveBot. No special action is required regarding these talk page notices, other than regular verification using the archive tool instructions below. Editors have permission to delete these "External links modified" talk page sections if they want to de-clutter talk pages, but see the RfC before doing mass systematic removals. This message is updated dynamically through the template {{source check}} (last update: 18 January 2022).

• If you have discovered URLs which were erroneously considered dead by the bot, you can report them with this tool.
• If you found an error with any archives or the URLs themselves, you can fix them with this tool.

Cheers.—InternetArchiveBot (Report bug) 22:34, 13 December 2016 (UTC)

Corrections on 0xE5 and 8" Floppies

8" Floppies did not come "CP/M Formatted", but rather were purchased as "IBM 3740 Formatted". Digital Research simply took advantage of the fact that IBM chose to fill sectors on track 2 (the CP/M directory) with EBCDIC "V" characters, and thus established the convention of 0xE5 meaning an empty CP/M directory entry. IBM 3740 Formatting had EBCDIC SPACE characters (0x40) in sectors on track 0, if I recall correctly, which I believe was where IBM had their directory. But most new CP/M formats (8" DD, 5.25", etc) tended to just fill all sectors with 0xE5 and avoid the need to know where the CP/M directory would be located. So, the 0xE5 originated from IBM mainframes but was perpetuated in the (pre-IBM) PC world by Digital Research (and perhaps others).

Durgadas311 (talk) 15:03, 1 March 2017 (UTC)

Feel free to make whatever corrections are appropriate, preferably with a citation. Guy Harris (talk) 16:50, 1 March 2017 (UTC)

Keeping this article current

Perhaps this article needs to be divided into Disk formatting and History of disk formatting assuming we can come to a consensus of what media and hardware is obsolete and add the appropriate headnote. patsw (talk) 18:03, 4 August 2018 (UTC)

Definition of high level formatting

The current definition of high level formatting is a restatement of the Tannebaum reference. The question is whether subsequent structures created within a file system constitute high level formatting as the phase is used in the art. An example of such structures are the many database formats used in various file systems see, e.g. Database Files which AFAIK construction of which is not in the art called high level formatting, usually it's just formatting. Accordingly I removed "containing file systems" from the definition. Comments? Tom94022 (talk) 22:14, 4 October 2021 (UTC)

A file system in the sense in which it's generally used can exist within a disk image file in another file system (the same file system type or a different file system type). For example, ISO images are used to install OSes on virtual machines, and macOS applications are often distributed in ".dmg files" containing an HFS+ file system.

That might be what "containing file systems" was intended to refer to. Guy Harris (talk) 22:21, 4 October 2021 (UTC)

A disk can be formatted with no file system, e.g., a CP owned disk in VM. Also, a file system can exist within another file system, e.g., a zFS in z/OS. I believe that the term high-level formatting applies to both of those cases. I wasn't considering the case of a disk image, e.g., ISO, file containing a file system. --Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:37, 5 October 2021 (UTC)

I don't think an ISO image of a disk is a file system, but is instead a file containing an image. Tom94022 (talk) 18:54, 6 October 2021 (UTC)

Not all "disk image" files are "disk images" in the sense that they're a byte-for-byte copy of a physical CD/DVD or a physical HDD/SSD. macOS, for example, has the Disk Utility GUI application and the hdiutil command-line utility, which can create an empty, partitioned disk image, which can then be "attached", so that the partitions appear as disk devices with /dev entries. You can then use any tool that can do "high-level formatting" of a real HDD/SSD to do "high-level formatting" of the "fake" disk; every write to the disk device gets forwarded to a daemon process that performs them by writing to a file. No disks involved except for the disks containing the file system the file is on. Guy Harris (talk) 21:27, 6 October 2021 (UTC)

I agree, but it doesn't seem to me that building such images amounts to "create[ing] the file system format within a disk partition or a logical volume" which is the Tanenbaum ref. The already created file system then uses the image as I understand it. Tom94022 (talk) 22:26, 13 October 2021 (UTC)

What do you mean by "file system" here? If an ISO image were copied byte-for-byte to a CD, would that CD contain a file system? If so, then presumably the ISO image contains a file system, too. If an Apple disk image file containing an HFS+ or APFS file system were copied byte-for-byte to a disk partition, would that disk partition contain a file system? If so, then presumably the disk image file contains a file system, too.

Now, if you want to say that writing file system data structures - "creating the file system format" - to a disk partition or logical volume constitutes "high-level formatting", but writing file system data structures - "creating the file system format" - within a file doesn't, that's different. Both "create the file system format", but "formatting" could, I guess, be restricted to operations that write directly to a disk or partition rather than writing to blocks of a file within an existing file system. Guy Harris (talk) 23:09, 13 October 2021 (UTC)

This article in the lede defines Disk formatting as Disk formatting is the process of preparing a data storage device ... for initial use. In some cases, the formatting operation may also create one or more new file systems. High level formatting is the third step in the process which I think restricts this to operations that write directly to a disk or partition rather than writing to blocks of a file within an existing file system. BTW the lede quote above supports including the IDKDSF as high level formatting even though IBM doesn't use the term explicitly. Tom94022 (talk) 19:52, 14 October 2021 (UTC)

A disk formatted with no file system has not been high level formatted in the common meaning of the term. In reviewing several relevant IBM publications I could find no reliable source describing the formatting of VSAM Linear Data Set (LDS) as high level formatting or that it was even a file systems so it seem that it is not a file system in the common meaning of that term but represents subsequent formatting within a file system (z/OS) that makes the formatted contents available to, in this example an IBM access method, but more generally this is a case of subsequent formatting within a file system that makes contents available to an application (or access method) as for example the many Database Files which must be formatted before a database can access them. I propose adding language covering subsequent formatting to the section to cover both recently the added language about VSAM Linear Data Set (LDS) and database files. Tom94022 (talk) 18:54, 6 October 2021 (UTC)

Then what would you call the process of writing R0 on every track? It's certainly not the sort of formatting that's done at the factory and does not match the description of low-level formatting in the article.

The LDS containing a zFS exists withinn a file system, so the zFS exists within a file system. --Shmuel (Seymour J.) Metz Username:Chatul (talk) 23:57, 6 October 2021 (UTC)

I would call the several processes that IBM describes in IDKDSF as high level formatting even though IBM doesn't. Specifically on p.74 for CKD devices and p. 363 for FBA devices IBM discloses "formatting" necessary "for your operating environment" using INIT, CPVOLUME or AIXVOL Commands depending upon the environment; to answer your question, the INSTALL command writes home address and record 0 on every track of a certain volumes. The problem is semantics in that IBM uses some language in different ways than the rest of the industry and I can't find anything that would lead me to conclude that "formatting an LDS that contains a zFS" amounts to creating a file system any more than creating a database file or files amounts to creating a file system. Tom94022 (talk) 22:26, 13 October 2021 (UTC)

Here is a dictionary definition:

Definitions of high-level formatting

(noun) (computer science) the format for the root directory and the file allocation tables and other basic configurations

type of:

data format, data formatting, format, formatting

the organization of information according to preset specifications (usually for computer processing)

Unless there is further discussion I am going to further revise the section to limit high level formatting to operations that write directly to a volume or partition rather than writing to blocks of a file within an existing file system and including only the IDKDSF commands and language as such for IBM. Tom94022 (talk) 19:16, 16 October 2021 (UTC)

A root directory and a file allocation table can exist within a plain file:

$ hdiutil create -megabytes 1.5 -fs MS-DOS /tmp/fatimage
created: /tmp/fatimage.dmg
$ file /tmp/fatimage.dmg 
/tmp/fatimage.dmg: DOS/MBR boot sector; partition 1 : ID=0xb, start-CHS (0x3ff,254,63), end-CHS (0x3ff,254,63), startsector 1, 3071 sectors, extended partition table (last)
$ hdiutil attach /tmp/fatimage.dmg 
/dev/disk2          	FDisk_partition_scheme         	
/dev/disk2s1        	DOS_FAT_32                     	/Volumes/NO NAME
$ echo "This is a file inside a FAT file system inside a file " >/Volumes/NO\ NAME/file.txt
$ ls /Volumes/NO\ NAME/
file.txt
$ grep 'This is a file' /Volumes/NO\ NAME/file.txt 
This is a file inside a FAT file system inside a file 
$ grep 'This is a file' /tmp/fatimage.dmg
Binary file /tmp/fatimage.dmg matches
$ grep 'This is not a file' /tmp/fatimage.dmg
$ ls -l /tmp/fatimage.dmg
-rw-r--r--@ 1 gharris  wheel  1572864 Oct 16 14:18 /tmp/fatimage.dmg
$ mount | egrep disk2
/dev/disk2s1 on /Volumes/NO NAME (msdos, local, nodev, nosuid, noowners, mounted by gharris)

The file {{mono}/tmp/fatimage.dmg}} is, as shown above, an ordinary plain file, i.e., this being a UNIX(R) (macOS Big Sur), it's a seekable array of bytes, just as /etc/passwd or /bin/cat or... are.

It also happens to contain a root directory and a file allocation table, as its contents aren't a text file giving user names, user IDs, etc. or a Mach-O file containing the executable code of the cat command or..., they're a FAT file system. It also contains, in that root directory, a file named file.txt, whose contents are the string "This is a file inside a FAT file system inside a file "; if I search for that string in the plain file /Volumes/NO NAME/file.txt, it shows up, and if I search for that string in the plain file /tmp/fatimage.dmg, it shows up.

And if I change the file in the FAT file system:

$ echo "This is truly a file inside a FAT file system inside a file" >/Volumes/NO\ NAME/file.txt
$ grep 'This is a file' /Volumes/NO\ NAME/file.txt
$ grep 'This is a file' /tmp/fatimage.dmg
$ grep 'This is truly a file' /Volumes/NO\ NAME/file.txt
This is truly a file inside a FAT file system inside a file
$ grep 'This is truly a file' /tmp/fatimage.dmg
Binary file /tmp/fatimage.dmg matches

the contents of the image file change.

The definition of "high-level formatting" given by vocabulary.com does not say anything about disks/drives or partitions within a disk/drive - which is a good thing, as there exists an object (/tmp/fatimage.dmg on my machine) that is not a disk/drive and is not a partition within a disk/drive but nevertheless contains a root directory and a file allocation table. It happens to contain partitions, just as a disk/drive can contain partitions, but it's not something in the physical form of a disk/drive, it's an object living within an APFS file system, whose contents are ultimately managed by APFS, with reads from and writes to the "disk/drive" going through diskimages-helper:

$ ps -ef | egrep diskimages-helper
  501 21431     1   0  2:41PM ??         0:00.03 /System/Library/PrivateFrameworks/DiskImages.framework/Resources/diskimages-helper -uuid EF438C6A-EFCF-4613-BF3F-707A47DA7994 -post-exec 4

So "further revis[ing] the section to limit high level formatting to operations that write directly to a volume or partition rather than writing to blocks of a file within an existing file system" is fine as long as it's acknowledged that "a volume or partition" might exist within a file within an existing file system, meaning that "[writing] directly to a volume or partition" could ultimately turn into "writing to blocks of a file within an existing file system". (Well, technically, in my example, the writing was done directly to a file within an existing file system, as it was done by hdiutil create, before the disk image file was attached by hdiutil attach. It might be that there's a way to create an empty file, attach it such that there's a device file in /dev for it and diskimages-helper implements reads from and writes to that device file as reads from and writes to the file, and then run newfs_msdos on that device file to put the root directory and file allocation table into the underlying file, and then mount that device file.) Guy Harris (talk) 21:51, 16 October 2021 (UTC)

The opening paragraph of the article describes three "formatting" steps - "low-level formatting", "partitioning", and "high-level formatting".

The latter two steps can, for most file systems, be done to anything that can be accessed as a seekable array of fixed-length blocks or, for "high-level formatting", to a partition on something to which "partitioning" has been done, so they can be done to a plain file on a "files are seekable arrays of bytes" file system (e.g. Unix and systems inspired by Unix in that regard, such as MS-DOS, Windows, etc.) by software that divides that file into fixed-length blocks, just as they can be done to a disk/drive that can be accessed as a seekable array of fixed-length blocks.

For CKD drives and file systems that use them, things may be different (although I have the impression that these days, a "CKD drive" is a virtual device implemented atop a drive that's a seekable array of fixed-length blocks).

(We leave out further layers of block virtualization here.)

So the only stuff in "disk formatting" that's specific to disks is low-level formatting. Guy Harris (talk) 22:22, 16 October 2021 (UTC)

For CKD, the underlying hardware is an array of sector-oriented volumes, and a volume seen by the channel may be spread across parts of multiple physical volumes. As seen by the OS, a file system is a seekable sequence of blocks that need not be of a uniform size, e.q., the directory of a PDS has fixed length (KL=8, DL=256) blocks while the members have different characteristics. --Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:29, 17 October 2021 (UTC)

There nothing in that definition that limits it to existing directly in a disk or partition; it applies equally to formatting a file system that exists in a file of a containing file system. --Shmuel (Seymour J.) Metz Username:Chatul (talk) 14:29, 17 October 2021 (UTC)

Sorry to have to repeat this, but above I pointed out that:

This article in the lede defines Disk formatting as "Disk formatting is the process of preparing a data storage device ... for initial use. In some cases, the formatting operation may also create one or more new file systems." High level formatting is the third step in the process which I think restricts this to operations that write directly to a disk or partition rather than writing to blocks of a file within an existing file system.

It seems to that all the formatting discussed by Chatul and Harris is additional formatting beyond that which was done to make a storage device initially available to the system. Such additional formatting is not high level formatting as defined by Tanenbaum and as stated in the leve. The UNIX example posted by Harris is a an example of additional formatting long after the storage medium was made available to the system by use of a very different set of commands such as fdisk, mkfs, format, etc see "how to install a new disk drive in UNIX". IDKDSF describes the IBM commands including its version for UNIX.

This really is a semantics problem - Disk formatting and high level formatting both have very specific common meanings which is the way this article is written. They are all about initialization! I would have no problem with an addition to the lede that after the disk is formatted, additional formatting can create specialized structures within a file system such as database files or even additional file systems or words to that effect, of course, with suitable reliable sources. But I suggest it is either OR or POV to expand the meaning of either disk formatting or high level formatting beyond their common usage as initializing storage media. Tom94022 (talk) 23:00, 17 October 2021 (UTC)

"It seems to that all the formatting discussed by Chatul and Harris is additional formatting beyond that which was done to make a storage device initially available to the system." The only formatting absolutely necessary to make a storage device initially available to the system, in the sense of the system being able to read from and write to the device, is low-level formatting. The storage device is then available for the system to do partitioning or high-level formatting on it. So, given that the formatting operations I was discussing were partitioning and high-level formatting, your statement is true - but not necessarily in the sense that you intended it to be true.

"The UNIX example posted by Harris is a an example of additional formatting long after the storage medium was made available to the system by use of a very different set of commands such as fdisk, mkfs, format, etc" Exactly. fdisk is an example of a command that does partitioning, and mkfs/newfs (and the file-system-type-dependent commands they run) are the commands that are used to create file systems on a UN*X system, including "high-level formatting" by running it on a storage medium rather than, for example, a plain file. It's not as if you have to have a storage medium that already has been partitioned by some tool other than, say, fdisk, and has had a file system put on it by some tool other than, say, mkfs/newfs, in order to use fdisk or mkfs/newfs.

So perhaps "high-level formatting" should be described as the process of writing a new file system to a storage medium, i.e. mkfs/newfs is a command to write a new file system and, when handed a special file for disk partition of a storage device as the file to which to write, it does "high-level formatting".

As for "partitioning", the article misrepresented it, saying it "[made] the data storage device visible to an operating system" and "[allowed] access by an operating system". That's not what the Tanenbaum reference said, and it's not clear what it meant. Part of what it does, as the name itself indicates, is to divide the disk into multiple partitions, i.e. sub-devices. Depending on the system and the way it boots, it might also involve writing information to allow an operating system to be booted from the medium, but that has nothing to do with "partitioning" per se. It may also be that some operating systems require there to be a partition table on a device in order to access file systems on the device, but that's not true of all operating systems (early UNIX systems had the partition table compiled into the device driver, and you could set up a file system on a partition that began at the first sector of the drive and ran to the end of the drive). And, given that "partitioning" is often done by programs running under an operating system, unpartitioned drives are accessible to the operating system in at least some sense.

I've cleaned it up somewhat, but it needs more work. I think the same utility partitions drives and adds boot information on most UN*Xes and on Windows, but, on IBM mainframe OSes, is there even a notion of "partitioning" a drive in the current UN*X/Windows sense, with the drive itself containing information about the partitions (rather than in the minidisk sense), and, if so, is that done by the same program that writes IPL information?

Once that's cleaned up, we can worry about what to call "partitioning" of an arbitrary object (including an object within a file system) and what to call that operating when it's specifically done to a data storage device. Guy Harris (talk) 02:20, 18 October 2021 (UTC)

It may help to note that there's a division between "low-level formatting" and all subsequent storage initialization operations.

"Low-level formatting" is device-type-specific and device-specific. On modern systems, it's not something that a host computer normally does, or perhaps even can do; it's done at the factory. As the article notes, at least some older systems can do low-level formatting. Low-level formatting isn't done to a logical volume, just a physical device. I'm not sure whether setting up bad-block maps, a flash translation layer, etc. would be the "low-level formatting" done for an SSD.

Low-level formatting to a directly-attached device is usually sufficient to allow the host to which the drive is attached (or to which the device's controller is attached) to directly read and write to the device; no subsequent formatting operations are necessary for that (given that the utilities that perform subsequent formatting operations run on the host and perform those read and write operations).

The utilities that operations performed after low-level formatting expect an abstract device that has a sequence of individually-accessible and randomly-accessible blocks. That abstraction might be provided by the disk controller driver, for devices whose native hardware addressing is cylinder/head/sector, with the driver performing the necessary division and modulo operations to convert between a block number to cylinder/head/sector; that driver might also perform bad-block mapping. Other devices, such as SCSI devices, do that in the controller.

In addition, a RAID controller, or a logical volume manager, might provide such an abstraction built atop the previous abstraction for devices directly attached to the host or RAID controller. A controller on a Fibre Channel network might do the same to devices attached to the Fibre Channel network for the benefit of hosts attached to that network.

In addition, a device on a Fibre Channel network might provide that abstraction on top of file-like containers within a file system on the device; that's what NetApp's appliances do, and there are probably other vendors who do the same. ("file-like" because they don't, as I remember, have file system directory entries pointing to them, but they are extensible variable-length containers just as files and directories are.)

And, on top of that, here's what the C: drive on a Windows machine I have looks like:

$ ls -ll ~/Documents/Virtual\ Machines/Windows\ 10\ Pro\ 64-bit.vmwarevm/Virtual\ Disk*.vmdk
-rw-r--r--  1 gharris  admin  4084006912 Oct 16 16:07 /Users/gharris/Documents/Virtual Machines/Windows 10 Pro 64-bit.vmwarevm/Virtual Disk-s001.vmdk
-rw-r--r--  1 gharris  admin  4261937152 Oct 16 16:07 /Users/gharris/Documents/Virtual Machines/Windows 10 Pro 64-bit.vmwarevm/Virtual Disk-s002.vmdk
-rw-r--r--  1 gharris  admin  4236443648 Oct 16 16:07 /Users/gharris/Documents/Virtual Machines/Windows 10 Pro 64-bit.vmwarevm/Virtual Disk-s003.vmdk
-rw-r--r--  1 gharris  admin  4253745152 Oct 16 16:07 /Users/gharris/Documents/Virtual Machines/Windows 10 Pro 64-bit.vmwarevm/Virtual Disk-s004.vmdk
-rw-r--r--  1 gharris  admin  4261937152 Oct 16 16:07 /Users/gharris/Documents/Virtual Machines/Windows 10 Pro 64-bit.vmwarevm/Virtual Disk-s005.vmdk
-rw-r--r--  1 gharris  admin  4261937152 Oct 16 16:07 /Users/gharris/Documents/Virtual Machines/Windows 10 Pro 64-bit.vmwarevm/Virtual Disk-s006.vmdk

...

That "disk" was partitioned, and had high-level formatting done on it by Microsoft's Windows 10 installer, probably operating under the impression that it was just writing to an ordinary hard drive (or SSD - the files in question reside on an SSD, but I don't know whether VMware pretends that the disk controller is running an HDD or an SDD in that case). I.e., other than the low-level formatting, that "disk" was probably initialized in the same way that a real disk would be initialized, by software that probably (except maybe at the driver level, if there's any paravirtualization being done) has no clue that it's initializing a VM's drive rather than a real drive.

So, even if we ignore disk images in the macOS sense, all steps after "low-level formatting" can be performed by writing to what the software doing the operations thinks is a disk, or at least look like a disk, even if it's made out of multiple disks, or out of file-like object, or out of one or more files, or....

I.e., the first paragraph in the article might claim that "Disk formatting is the process of preparing a data storage device such as a hard disk drive, solid-state drive, floppy disk or USB flash drive for initial use.", but sections 2.3 and 2.4 of the article often apply to a data storage "device" that's more complicated than an HDD/SDD/FDD (a "USB flash drive" jut being a USB SSD). Those operations can be applied to a single device, but there's no guarantee that they are being applied to a single device as opposed to some more complicated abstract "device".

(And maybe something other than a textbook from 2001 is required here, as 1) some of the technologies I mentioned that implement those abstractions were either fairly new in 2001 or hadn't yet been implemented in 2001 and 2) textbooks intended for undergraduates simplify the topic - its description of "partitioning" is quite PC-specific (and talks about "Pentium"s, probably to further simplify the topic for newbies), and its description of "high-level formatting" omits the "install the operating system on the shiny new file system" step of arranging that "at this point the system can be booted".)

So I'd say that, unless the number of times when a RAID array/virtualized device on an FC network/virtualized device provided by a hypervisor is being partitioned and one or more partitions are having file systems written to them is negligible (even when taking into account that there are a lot of big servers out there, even if most larger-then-tablet computers aren't servers - i.e., servers are important enough that I don't think they can just be dismissed), I'd say that the article should speak of partitioning and "high-level formatting" being done to a "volume" or some such term that doesn't imply that it's a single physical HDD or SDD.

And perhaps we need to draw a distinction between "partitioning" and "making a volume bootable", as 1) "partitioning" isn't sufficient to make a volume bootable, as, for example, an MBR volume with a boot block and a bunch of empty file systems doesn't have anything to boot and 2) "partitioning" isn't always necessary to make a volume bootable, as, at least for some OSes, it might be possible to just put a file system containing a bootable OS directly on an unpartitioned disk.

As for a zFS, is it ever the case that a DASD is initialized with zFS and no other file system on it? If not, then perhaps it could be argued that installing a zFS, if not done as part of the process of initializing a volume for use on z/OS, might not count as "high-level formatting" if "high-level formatting" is defined as the "set up a new file system" process when done on a volume (in the sense of "something that a bunch of firmware and software, running on some combination of the host computer and possibly devices on a network, makes look something like a disk". Guy Harris (talk) 06:24, 18 October 2021 (UTC)

Partitioning need not make a partition bootable; a partition can be data only. On PC-DOS and OS/2 there is a maximum of 4 partitions, one of which can be an extended logical partition and itself be partitioned. As an example, on my main PC I have 5 bootable partitions and 7 data-only partitions, most of which are logical drives in the extended logical partition. ArcaOS, a rebranded OS/2, normally is installed on JFS, which requires that IBM's Logical Volume Manager (LVM) do the partitioning.

ICKDSF initializes a volume with a VTOC, at which point z/OS can allocate a legacy dataset on the volume but cannot store a Unix file on it. In order to store a Unix file on the volume, the installation must create a file system on the volume and mount that file system. --Shmuel (Seymour J.) Metz Username:Chatul (talk) 03:48, 19 October 2021 (UTC)

If ...something other than a textbook from 2001 ... can be found then the article could be expanded to include much of the dicta in this talk but until there is a reliable source for a different definition of Disk formatting and its three steps I suggest this article is bound by that reference. Frankly I doubt if there is a more "modern" definition. FWIW, I have looked for one and not found any and what I have found supports the current article which basically limits the terms to those three steps in preparing a physical storage medium for initial use for which the specific OS commands for step 3, "high-level formatting" are identified hereinabove (e.g., fdisk, mkfs, format, INIT, CPVOLUME, etc.). Everything else is just subsequent formatting and beyond the current scope of the article. I suppose we could expand the scope of the article by adding a section on "Subsequent formatting" to add as much as we want but until there is an RS for a different common meaning to the term and its components the article should not change the meaning of the terms. 18:49, 23 October 2021 (UTC)

Notes